--- layout: page title: Lab 3 - Non-Compliance permalink: /labs/lab3/ parent: Labs nav_order: 3 ---
In the lecture we discussed selection bias, which is the difference in the potential untreated outcomes between those who were treated and who were not treated. Stated formally:
\[E[Y_{1i} | D_i = 1] - E[Y_{0i} | D_i = 0] = \underbrace{E[Y_{1i} - Y_{0i} | D = 1]}_{ATT} + \underbrace{E[Y_{0i} | D_i = 1] - E[Y_{0i} | D_i = 0]}_\text{Selection bias}\] Using experiments, we can get rid of selection bias as under randomisation units across treatment arms are - in expectation - similar on all (un)observed traits and only differ in terms of treatment assignment \(D_I\).
Given that, in expectation potential outcomes are orthogonal (independent) to treatment assignment: \(E[Y_{1i} | D_i = 1] = E[Y_{1i}]\) and also \(E[Y_{0i} | D_i = 0] = E[Y_{0i}]\)
That is, pre-treatment differences in potential outcomes are, on expectation, equal between units assigned to the treatment and the control group. Stated formally: \[E[Y_{0i} | D_i = 1] = E[Y_{0i} | D_i = 0]\] Using both of these assumption, the ATT is equivalent to the ATE. Stated formally:
\[ ATE = E[Y_{1i} | D_i = 1] - E[Y_{0i} | D_i = 0] = \underbrace{E[Y_{1i} - Y_{0i} | D = 1]}_{ATT} + \underbrace{E[Y_{0i} | D_i = 1] - E[Y_{0i} | D_i = 0]}_\text{0}\]
We also discussed ways that we randomly allocate units to either treatment or control:
Simple random assignment: a procedure such a die roll or coin toss that gives each subject identical probability of being assigned to treatment.
Complete random assignment: where m of N units are assigned to the treatment.
Stratified or block randomisation: We conduct complete randomisation within selected groups or within blocks, respectively. Blocks should be determined by a pre-treatment covariate that predicts outcomes.
Often times, we assign units to the treatment group, but they are not eventually treated. Usually, the raeason for not taking the treatment is correlated with the outcome. Thus, we need to figure out how to estimate the treatment effect taking into account that we may have either one-sided or two-sided non compliance.
Naive solution: Exclude those not treated, and calculate the different-in-means using only treated units. However, if we do this randomisation is broken.
Intention to treat (ITT): Difference in outcomes between units assigned to the treatment group versus those in the control group without taking into account compliance rates.
\[ ITT = E[Y_{1i}] - E[Y_{0i}] \]
\[ CACE = \frac{ITT}{E[D_{1i}]} \] where \(E[D_{1i}]\) is the difference of treatment compliance rates between the treated and the control group.
Before starting this seminar
Create a folder called “lab3”
Download the data (you can use this button or the one at the top, or read csv files directly from github):
Open an R script (or Markdown file) and save it in our “lab3” folder.
Set your working directory using the setwd() function or by clicking on “More“. For example setwd(“~/Desktop/Causal Inference/2024/Lab3”)
Let’s install an load packages that we will be using in this lab:
library(jtools) # generate plots with regression coefficients
library(stargazer) # generate formated regression tables
library(modelsummary) # generate formatted regression tables
library(texreg) # generate formatted regression tables
library(nnet) # If you want to use glm() to check for balance
library(haven) # upload .dta files into a R environment
library(tidyverse) # to conduct some tidy operations
library(ggplot2) # to generate plots
library(AER) # to run IV regressions
library(formatR) # using tidy
library(ggstance) # even more plots
In this seminar, we will cover the following topics:
lm() function to retrieve the ATE.stargazer package and
coefficients using the coefplot packageIntention-to-treat using the
mean(), lm(), ivreg()
functions.mean() function and Two-Stage-Least Squares using the
ivreg() function.You already know various types of experiments, such as survey experiments, lab experiments, field experiments.
How do natural experiments fit in?
Today We will be working with data from Susan Hyde’s work on the Observer
Effect in International Politics, where she analyses a
natural experiment in Armenia.
Hyde sought to identify a relatively broad but surprisingly
understudied research question: Do international monitoring missions
have an effect on electoral outcomes?
Satisfactorily answering this question is harder than it seems. As
Hyde reasons, cross-sectional comparisons (i.e. comparing mutliple
countries) as well as cross-temporal comparisons cannot provide robust
evidence of an observer effcect. Endogeneity will inevitably play a role
in such analyses. She finds a solution to this problem by observing the
microlevel. Instead of looking at outcomes at national level,
she wants to find out what effect observers have on election-day fraud.
To do so, she analyses individual polling stations.
Hyde selects the 2003 presidential elections in Armenia for her
analysis, for three particular reasons:
There clearly was some fraud on election day (remember: we need
variation in the outcome). The election has been described as ‘one
of the dirtiest even Armenians can remember’ (p.47).
Disaggregated data were available (note that this was by no means
a given in 2003). [Do you think reporting election results online
becoming a standard in virtually all countries has impacted fraud? Why
(not)?]
Assignment was ‘as-if’ random by the OSCE. Hyde mentions some
possible probems associated with the randomness of observer assignment
but argues that these were mitigated by the OSCE’s approach.
Given the assumption of random assignment, Hyde conceives of the
assignment of observers as natural experiment. The outcome of interest
is the incumbent’s vote share as the incumbent candidate is likely to
use the administrative apparatus to commit fraud. The assumption of
randomness entails that the expected outcome of both observed and
unobserved polling stations would be the same if no observer mission
took place.
Hyde concludes that the observer effect is signifcant. That is, the
presence of international observers on election day prevents fraud in
the observed polling stations. She finds that between observed and
unobserved polling stations there was a 5.9% difference
in incumbent vote share in the first round and a 2%
difference in the second round. Note that these are simple differences
in means.
A description of the variables is listed below:
| Variable | Description |
|---|---|
mon_voting |
Binary variable equal to 1 if observers were present in polling station in Round 1 |
kocharian |
Vote share of the incumbent (Robert Kocharian) in Round 1 |
mon_voting_R2 |
Binary variable equal to 1 if observers were present in polling station in Round 2 |
KocharianR2 |
Vote share of the incumbent (Robert Kocharian) in Round 2 |
osce_assignment |
Binary variable equal to 1 if OSCE was assigned to observe the polling station (Note: these are fictional data) |
pollingstation |
Name and number of polling station |
subregion |
Armenian region the polling station is located |
urban |
Binary variable equal to 1 if the polling station is located in an urban area |
voterspersqkm |
Continuous variable indicating population density |
Now let’s load the data. There are two ways to do this:
You can load the brands dataset from your laptop using the
read_dta() function. Remember to load the
haven package. If you stored your data in the same
directory that set your working directory, you only need to write the
name of the dataset, plus the extension (.dta).
# setwd("C:/Desktop/Causal Inference/2024/Lab3")
#
# library(haven)
# hyde2007 <- read_dta("hyde2007.dta") # loading your data from your laptop
Or you can load the data from the course website using the
read.dta() function. Write the following url: https://widening-sarajevo.github.io/CI/data/hyde2007.dta?raw=true
inside the read.dta() function
hyde2007=read_dta("https://widening-sarajevo.github.io/CI/data/hyde2007.dta?raw=true")
# updating the file from the course website.
Let’s start this lab by inspecting the data. You can use either
head() or glimpse() function. You may want to
select the variables that you want to inspect and then combine these
variables using the cbind() function. The syntax would be
the following
head(cbind(dataframe$variable1, dataframe$variable2...dataframe$variable3)).
Exercise 1: Inspect the following variable: pollingstation, urban,voterspersqkm,kocharian,mon_voting,mon_voting_R2, KocharianR2, osce_assignment.
head(cbind(hyde2007$pollingstation, hyde2007$urban, hyde2007$voterspersqkm, hyde2007$kocharian, hyde2007$mon_voting, hyde2007$mon_voting_R2, hyde2007$KocharianR2, hyde2007$osce_assignment))
[,1] [,2] [,3]
[1,] "\xa0 1671. Agarak, Club" "0" "27.9790992736816"
[2,] "\xa0 1620. Arevis, School" "0" "77.8955993652344"
[3,] "\xa0 0691. Urtsalanj, School" "0" "88.6297988891602"
[4,] "\xa0 0959. Norabak, secondary School" "0" "41.4747009277344"
[5,] "\xa0 1188. Shamlugh, Cultural House" "0" "63.1679000854492"
[6,] "\xa0 0678. Erasghavan, School" "0" "88.6297988891602"
[,4] [,5] [,6] [,7] [,8]
[1,] "1" "1" "0" "0.977800011634827" "1"
[2,] "0.942900002002716" "0" "0" "1" "1"
[3,] "0.529999971389771" "1" "0" "0.676199972629547" "1"
[4,] "0.693899989128113" "1" "0" "0.868399977684021" "1"
[5,] "0.506200015544891" "0" "0" "0.874300003051758" "1"
[6,] "0.685899972915649" "1" "0" "0.776600062847137" "1"
Now let’s check how many observations were observed or
treated. The treatment variable is mon_voting. We
can use the table() function to get the number of polling
stations were monitored. The syntax is the following:
table(df$variable):
Exercise 2: How many polling stations were observed in the first round?
table(hyde2007$mon_voting)
0 1
1016 748
We can see that out of the 1764 polling
stations,748 were monitored.
Now that we have inspected the data, let’s calculate the difference
in means of Kocharian’s vote share (kocharian) between
monitored and not monitored polling stations. Let’s check if we get the
same numbers reported by Hyde.
Exercise 3: Calculate the difference in means using the
mean() function. Remember to subset the data by setting a
logical condition within the squared brackets
[df$variable == TRUE, ]. The data frame will subset the
data selecting the rows that meet the logical condition. Here is the
syntax to calculate the difference in means
mean(df$outcome[df$treatment == 1]).
diffmeans <- mean(hyde2007$kocharian[hyde2007$mon_voting==1]) - mean(hyde2007$kocharian[hyde2007$mon_voting==0], na.rm=T)
diffmeans
[1] -0.05867601
We got the same estimates as Hyde. This difference in means that can
be interpreted as an average difference in vote share of
5.9 percentage points in support of Kocharian in polling
station that were monitored versus those not monitored.
We can also get the difference in means using the
t.test() We can do this by retrieving the means of each
groups from a two sample t-test output.
Exercise 4: Perform a two sample t-test using the
t.test() function. The syntax is as follows:
t.test(df$outcome ~ df$treatment). Use the assignment
operator <- to store the output of the t-test into an
object, which allows us to retrieve the means of each group from this
test.
t.test1 <-t.test(hyde2007$kocharian ~ hyde2007$mon_voting)
t.test1
Welch Two Sample t-test
data: hyde2007$kocharian by hyde2007$mon_voting
t = 5.9911, df = 1675.8, p-value = 2.546e-09
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
0.03946639 0.07788563
sample estimates:
mean in group 0 mean in group 1
0.5419031 0.4832271
To calculate the difference in means we need to retrieve the means of
both monitored and not monitored groups from the t-test output. To do
this, we need to know the names that the t.test() function
gives to each parameter that it generates. To do that, we can use the
names() function. The syntax is as follows:
names(object) .
names(t.test1) # we can see the names of each of the parameters generated by the t.test function.
[1] "statistic" "parameter" "p.value" "conf.int" "estimate"
[6] "null.value" "stderr" "alternative" "method" "data.name"
We see that the t.test() function stores the means and
calls them “estimate” .
We can retrieve the means from the object where we stored the t-test
output by including a dollar sign $, plus estimate. We can also use the
round() function to round off values to a specific number
of decimal values. The syntax of the round() function is
the following round(x, digits = 2) where x is the object
(variable). The second argument is the number of decimals that you want
to report.
Exercise 5: Obtain the mean vote share for Kocharian in
observed and unobserved polling stations. Use the output of the t.test
using the following syntax: t.test.object$estimate.Then,
retrieve the means of the monitored and not monitored polling stations
using the following syntax: t.test.object\$estimate[2] (for
monitored polling stations) and t.test.object\$estimate[1]
(for unmonitored polling stations). Finally, subtract these two objects
and round their difference, reporting 3 decimals. Use the
round() function to do this.
t.test1$estimate
mean in group 0 mean in group 1
0.5419031 0.4832271
diff_means_ttest <- round(t.test1$estimate[2] - t.test1$estimate[1], 3) # w
diff_means_ttest
mean in group 1
-0.059
We see that we obtained the same result as before:
-0.059. As you can see, this is not the most
straightforward way to retrieve the difference in means.
We know that we use regressions to analyse data from randomised experiments. It turns out that regressing an outcome variable on a treatment variable yields a slope coefficient identical to the difference in means between two groups. In addition, the resulting intercept corresponds to the average outcome among the control (untreated) units. Bear in mind that the difference in means estimator is equal to the coefficient of the treatment variable only when the treatment variable is binary.
Exercise 6: Regress the outcome variable
kocharian on the treatment variable,
mon_voting. Interpret the coefficient of the
mon_voting variable. Is the coefficient identical to the
difference in means obtained before? Load the stargazer
package and report this model in a regression table. The syntax of the
stargazer model is the following
stargazer(model, title="Results", align=TRUE, type = "text")
where model is the object where you stored the output of your
regression. Add a title to your regression table and call it “Results”.
Also set the align argument equal to TRUE, so the numeric
values in the same column will be aligned at the decimal mark. Finally,
set the type argument equal to "text" to
specifies what type of output the command should produce. You can also
use modelsummary() or texreg() to report
regression tables, feel free to use these packages as well.
library(stargazer)
ols1=lm(kocharian ~ mon_voting, data=hyde2007)
stargazer(ols1, title = "Results", align = TRUE, type = "text") # using stargazer
Results
===============================================
Dependent variable:
---------------------------
kocharian
-----------------------------------------------
mon_voting -0.059***
(0.010)
Constant 0.542***
(0.006)
-----------------------------------------------
Observations 1,764
R2 0.019
Adjusted R2 0.019
Residual Std. Error 0.206 (df = 1762)
F Statistic 35.022*** (df = 1; 1762)
===============================================
Note: *p<0.1; **p<0.05; ***p<0.01
From the regression we obtained the same value as when we “manually”
calculate the difference in means before. The beta coefficient of the
mon_voting variable means that the vote share for Kocharian
decreased by 5.9 percentage points once international
observers monitor the election process. We also see that the intercept
is equal to 0.542. This parameter represents the average
vote share for Kocharian in those polling stations that were not not
monitored.
As we did last week, we would like to look check that the randomisation was successful. In particular, we would like to look at whether there is balance between urban and rural areas.
Exercise 7: Create a contingency table using the
tab() function. Get the number of polling stations that
were monitored versus those that were not monitored (variable:
mon_voting), both for urban and rural areas (variable:
urban). The syntax is the following:
table(df$variable_1, df$variable_2). Are there differences
in the number of polling stations monitored between these two
areas?
table(hyde2007$urban, hyde2007$mon_voting)
0 1
0 500 285
1 516 463
We can see that there are more polling stations were monitored in
urban areas (463) versus in rural areas
(285).
Now, let’s use prop.table() to check whether there is
unbalance between urban and rural areas. You can do this by storing the
previous table in a object, and then insert this object inside the
prop.table() function.
Exercise 8: Run the prop.table() function. Do
you find unbalance in terms of the number of polling stations between
urban and rural?
prop.test(table(hyde2007$urban, hyde2007$mon_voting))
2-sample test for equality of proportions with continuity correction
data: table(hyde2007$urban, hyde2007$mon_voting)
X-squared = 21.088, df = 1, p-value = 4.388e-06
alternative hypothesis: two.sided
95 percent confidence interval:
0.06279495 0.15695353
sample estimates:
prop 1 prop 2
0.6369427 0.5270684
From the prop.table() function, we find that around
63.6% of the polling stations in the rural area were not monitored,
whereas in the urban area roughly 52%. We find a statistically
significant difference in the proportions of polling stations that were
not observed.
We can also check for balance by regressing the treatment assignment
variable mon_voting on the covariate(s) of interest
urban. Let’s do that in the following exercise.
Exercise 9: Regress mon_voting on
urban. Again, report the results in a regression table.
Change the title to “Randomisation check”.
balance_urban=lm(mon_voting ~ urban, data=hyde2007)
stargazer(balance_urban, title = "Randomisation check", align = TRUE, type = "text") # using stargazer
Randomisation check
===============================================
Dependent variable:
---------------------------
mon_voting
-----------------------------------------------
urban 0.110***
(0.024)
Constant 0.363***
(0.018)
-----------------------------------------------
Observations 1,764
R2 0.012
Adjusted R2 0.012
Residual Std. Error 0.491 (df = 1762)
F Statistic 21.777*** (df = 1; 1762)
===============================================
Note: *p<0.1; **p<0.05; ***p<0.01
What do we conclude? Do we have reason to think this might affect the
validity of the results? One key feature of randomised experiments is
that they might be able generate unbiased estimates of the average
treatment effect even if there is imbalance. We also know that we can
use the covariate urban to eliminate observed differences
between treatment and control groups and reduce variance in outcomes.
However, we should be suspicions that we find such a large
difference between the two areas areas.
Let’s ignore the differences in the number of polling stations
monitored between urban and rural areas for now. Let’s look at
heterogeneous treatment effects and identify whether there is a
differential response to international monitoring between urban and
rural areas. For that, we could interact the treatment variable
mon_voting with the urban variable. The syntax
is the following
lm(Outcome variable = Treatment variable + Covariate + Covariate * Treatment variable)
Exercise 10: Run a regression where the dependent variable is
voter share kocharian is regressed on
mon_voting, the urban variable and an
interaction between mon_voting and urban
variable. Interpret the coefficient of the mon_voting and
the interaction. Do you find evidence of a differential response to
being monitored between polling stations in urban versus rural areas?
Also, interpret the coefficient of the urban variable.
Present the results in a regression, but also include the previous model
conducted in Exercise 6 (bivariate model), so you can compare both
models. The syntax using stargazer is the following stargazer(model1,
model2, title = “Models, align =TRUE, type =”text”).
ols1_interaction =lm(kocharian ~ mon_voting + urban + mon_voting*urban, data=hyde2007)
stargazer(ols1, ols1_interaction, title = "Models", align = TRUE, type = "text")
Models
=====================================================================
Dependent variable:
-------------------------------------------------
kocharian
(1) (2)
---------------------------------------------------------------------
mon_voting -0.059*** -0.028*
(0.010) (0.015)
urban 0.003
(0.013)
mon_voting:urban -0.050**
(0.020)
Constant 0.542*** 0.540***
(0.006) (0.009)
---------------------------------------------------------------------
Observations 1,764 1,764
R2 0.019 0.025
Adjusted R2 0.019 0.023
Residual Std. Error 0.206 (df = 1762) 0.205 (df = 1760)
F Statistic 35.022*** (df = 1; 1762) 14.863*** (df = 3; 1760)
=====================================================================
Note: *p<0.1; **p<0.05; ***p<0.01
First, we find that controlling for urban the point
estimate of the mon_voting yields a much smaller treatment
effect (roughly 2 times smaller). Now the ATE is -0.028 but
yet statistically significant. The non-zero coefficient on the
interaction between urban and mon_voting means
that support to being monitored has a differential effect on vote share
for Kocharian between urban vs rural areas. This interaction coefficient
(0.050) is the average difference in the effect of
mon_voting between urban and rural areas. This
substantively means that the average effect of monitoring is associated
with a 5%point decrease in vote share for Kocharian, when me move from a
rural to urban area. Finally, the coefficient from the
urban variable indicates the average difference in vote
share for Kocharian between urban and rural areas, when there is no
monitoring mon_voting = 0. Here, we don’t find a
significant effect. This is unexpected as urban areas usually show
different voting patterns than rural regions do. Given the imbalanced
patterns of observation, the monitoring effect is likely to absorb the
rural-urban difference. This is a problem.
In this study, the OSCE (Organization for Security and Co-operation in Europe) conducted a second wave of inspections for the second round of the elections. Let’s quickly replicate the same analyses that we did before.
Exercise 11: Run a regression using the outcome variable
KocharianR2 regressed on the treatment assignment variable
mon_voting_R2. You can use the summary() or
tidy() function to report your results. Is the treatment
effect weaker or stronger in this round?
ols2 <- lm(KocharianR2 ~ mon_voting_R2, data=hyde2007)
summary(ols2)
Call:
lm(formula = KocharianR2 ~ mon_voting_R2, data = hyde2007)
Residuals:
Min 1Q Median 3Q Max
-0.69251 -0.10281 -0.00287 0.11904 0.32733
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.692511 0.004781 144.841 <2e-16 ***
mon_voting_R2 -0.019841 0.008043 -2.467 0.0137 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.1614 on 1761 degrees of freedom
(1 observation deleted due to missingness)
Multiple R-squared: 0.003444, Adjusted R-squared: 0.002878
F-statistic: 6.085 on 1 and 1761 DF, p-value: 0.01373
# tidy format
ITT <- tidy(ols2) %>%
filter(term == "mon_voting_R2") %>%
pull(estimate)
ITT
[1] -0.01984091
This looks like a somewhat weaker effect of the observers. We find
that the average treatment effect is now -0.02, which means
that there is a drop of 2 percentage points in the vote
share for Kocharian in the second round of elections when polling
stations are being monitored.
Exercise 12: Again, as we did before, let’s check if there is
balance for the urban variable in the second round. Regress
the treatment assignment variable mon_voting_R2 on
urban. Report the results using either
summary() or the tidy() function.
ols2_urban=lm(mon_voting_R2 ~ urban, data=hyde2007)
summary(ols2_urban)
Call:
lm(formula = mon_voting_R2 ~ urban, data = hyde2007)
Residuals:
Min 1Q Median 3Q Max
-0.4147 -0.4147 -0.2764 0.5853 0.7236
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.27643 0.01689 16.365 < 2e-16 ***
urban 0.13828 0.02267 6.099 1.31e-09 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.4733 on 1762 degrees of freedom
Multiple R-squared: 0.02067, Adjusted R-squared: 0.02012
F-statistic: 37.19 on 1 and 1762 DF, p-value: 1.313e-09
We find again unbalance between urban and rural areas. This linear
probability model suggest that being in a urban area is associated with
an increase on the probability of being observed by 13%
more than an rural area.
Given that we are working with a binary treatment variable, we could
also use glm(). You can try this on your own and interpret
the results from this estimation.
Let’s perform the same analysis that we did before for the first
round to obtain the ATE for the second round. Include the variable
urban and also include interaction between
urban and mon_voting_R2.
Exercise 13: Find the treatment effect using the
lm() function. Include the urban variable as a
covariate. Also include the interaction between
mon_voting_R2 with urban. Interpret the
results for both mon_voting_R2 an the interaction term.
Report your results in a regression table. Use either
stargazer(), screenreg() or
modelsummary(). Give the following names to your model:
“Second round”. Report the results with 3 decimals.
ols2_interaction=lm(KocharianR2 ~ mon_voting_R2 + urban + urban*mon_voting_R2, data=hyde2007)
library(texreg)
screenreg(ols2_interaction, digits = 2, custom.model.names = c("Second round"),
custom.coef.names = c("Intercept", "Monitor", "Urban", "Monitor:Urban"))
===========================
Second round
---------------------------
Intercept 0.67 ***
(0.01)
Monitor 0.01
(0.01)
Urban 0.04 ***
(0.01)
Monitor:Urban -0.06 ***
(0.02)
---------------------------
R^2 0.01
Adj. R^2 0.01
Num. obs. 1763
===========================
*** p < 0.001; ** p < 0.01; * p < 0.05
We find that the treatment effect for the second round is
0.01 - slightly smaller and no longer statistically
significant. However, we do find a differential response to the
treatment as the interaction is statistically significant. This
coefficient of the interaction can be interpreted as follows: The
average effect of monitoring on vote share for Kocharian is about
6 percentage points less, when we move from polling
stations in rural to urban areas.
Let’s include a in a single regression table some of the regressions
that we have conducted so far. You can use the stargazer()
function, but feel free to use texreg() or
modelsummary().
Exercise 14: Include the regression models into single
regression table. Change the title to: “Regression results - All
rounds”. Call bivariate models “Baseline” and models with interaction
“Interaction”. To do this, include the following argument
column.labels = c("Name 2", "Name3"). Also, set labels for
the dependent variables that you analysed the first round and the second
round. You can do this by adding the following argument
dep.var.labels = c("First Round", "Second Round"). Finally,
let’s change the labels of the variables by including the
covariate.labels = c("label variable 1", "label variable 2")
stargazer(ols1, ols1_interaction, ols2, ols2_interaction, title = "Regression results - All rounds", column.labels = c("Baseline", "Interaction", "Baseline", "Interaction"), dep.var.labels = c("First round", "Second round"), covariate.labels = c("Monitored", "Urban", "Monitor:Urban", "Monitor round 2:Urban", "Monitor round 2" ), align = TRUE, type = "text")
Regression results - All rounds
======================================================================================================================
Dependent variable:
------------------------------------------------------------------------------------------------
First round Second round
Baseline Interaction Baseline Interaction
(1) (2) (3) (4)
----------------------------------------------------------------------------------------------------------------------
Monitored -0.059*** -0.028*
(0.010) (0.015)
Urban 0.003 0.038***
(0.013) (0.010)
Monitor:Urban -0.050**
(0.020)
Monitor round 2:Urban -0.060***
(0.017)
Monitor round 2 -0.020** 0.013
(0.008) (0.013)
Constant 0.542*** 0.540*** 0.693*** 0.673***
(0.006) (0.009) (0.005) (0.007)
----------------------------------------------------------------------------------------------------------------------
Observations 1,764 1,764 1,763 1,763
R2 0.019 0.025 0.003 0.014
Adjusted R2 0.019 0.023 0.003 0.012
Residual Std. Error 0.206 (df = 1762) 0.205 (df = 1760) 0.161 (df = 1761) 0.161 (df = 1759)
F Statistic 35.022*** (df = 1; 1762) 14.863*** (df = 3; 1760) 6.085** (df = 1; 1761) 8.210*** (df = 3; 1759)
======================================================================================================================
Note: *p<0.1; **p<0.05; ***p<0.01
From the previous analysis we might wonder to what extend the rounds are independent of each other. Hyde states the following in this regard:
Let’s see whether monitoring in the two rounds was independent. First, we check the number of polling stations observed per round.
sum(with(hyde2007, mon_voting==0 & mon_voting_R2==0))
[1] 756
sum(with(hyde2007, mon_voting==1 & mon_voting_R2==0))
[1] 385
sum(with(hyde2007, mon_voting==0 & mon_voting_R2==1))
[1] 260
sum(with(hyde2007, mon_voting==1 & mon_voting_R2==1))
[1] 363
Well, in Round 2 more stations were observed that had been monitored in Round 1 than those that had not been monitored. Let’s confirm significance. For that we can conduct a two sample test of proportions.
prop.test(table(hyde2007$mon_voting_R2, hyde2007$mon_voting))
2-sample test for equality of proportions with continuity correction
data: table(hyde2007$mon_voting_R2, hyde2007$mon_voting)
X-squared = 98.233, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
0.1965443 0.2939381
sample estimates:
prop 1 prop 2
0.6625767 0.4173355
What do we conclude?
The evidence suggest that monitoring in round 2 was not in fact random - it was conditional on the results of the first round.
If we apply experimental standards (which is what we should do), we conclude that Round 2 monitoring was not independent from Round 1 monitoring.
Finally, let’s plot all coefficients from the regressions conducted
in Exercises 6, 10, 11 and 13 together. There are
multiple packages to plot regression estimates. One of them is
jtools. We can use the plot_summs function to
create regression coefficient plots with ggplot2.
The first step is to store all regressions into separate objects. You should have 4 models in 4 separate objects: two bivariate models and two models with interactions. We can store all of the objects in a list. A list in R is object that can contain multiple objects. This facilitates the syntax that we need to use whenever a plot with multiple models.
# bivariable model first round
ols1
Call:
lm(formula = kocharian ~ mon_voting, data = hyde2007)
Coefficients:
(Intercept) mon_voting
0.54190 -0.05868
# model with interaction first round
ols1_interaction
Call:
lm(formula = kocharian ~ mon_voting + urban + mon_voting * urban,
data = hyde2007)
Coefficients:
(Intercept) mon_voting urban mon_voting:urban
0.540319 -0.027811 0.003119 -0.050424
# bivariate model second round
ols2
Call:
lm(formula = KocharianR2 ~ mon_voting_R2, data = hyde2007)
Coefficients:
(Intercept) mon_voting_R2
0.69251 -0.01984
# model with interaction second round
ols2_interaction
Call:
lm(formula = KocharianR2 ~ mon_voting_R2 + urban + urban * mon_voting_R2,
data = hyde2007)
Coefficients:
(Intercept) mon_voting_R2 urban
0.67343 0.01324 0.03802
mon_voting_R2:urban
-0.05951
models <- list(ols1,
ols1_interaction,
ols2,
ols2_interaction)
The second step is to include this list the plot_summs()
function. Add the following argument inside of the
plot_summs() function, scale = TRUE. This generates a
version of the plot. Also, include the following argument:
robust = TRUE. Setting robust = TRUE the plot
will generate confidence intervals using robust standard errors.
Exercise 15: Generate a plot with the coefficients of all four models using `plot_summs() function
coefficients <- c("Intercept" = "(Intercept)","Monitor 1" = "mon_voting", "Urban" = "urban", "Monitor 1 x Urban" = "mon_voting:urban", "Monitor 2" = "mon_voting_R2", "Monitor 2 x Urban" = "mon_voting_R2:urban")
plot_summs(models, coefs = coefficients, scale = TRUE, robust = TRUE)
We already found meaningful imbalance. Polling stations in urban
areas were much more likely to be observed than rural ones in the first
round of the elections. We know that we can’t control for all possible
confounders, but is this imbalance a problem for the validity
of Hyde’s research design? Why?
Let’s check if we get a similar result for population density to make
sure the imbalance is not simply caused by the way the
urban variable was constructed.
Exercise 16: Find out whether randomization worked regarding
population density (voterspersqkm).
#Population Density
ks.test(hyde2007$voterspersqkm[hyde2007$mon_voting==0],
hyde2007$voterspersqkm[hyde2007$mon_voting==1])
Two-sample Kolmogorov-Smirnov test
data: hyde2007$voterspersqkm[hyde2007$mon_voting == 0] and hyde2007$voterspersqkm[hyde2007$mon_voting == 1]
D = 0.26816, p-value < 2.2e-16
alternative hypothesis: two-sided
# Lookin at means only
t.test(hyde2007$voterspersqkm~hyde2007$mon_voting, data=hyde2007)
Welch Two Sample t-test
data: hyde2007$voterspersqkm by hyde2007$mon_voting
t = -12.156, df = 1221.9, p-value < 2.2e-16
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-914.4994 -660.3231
sample estimates:
mean in group 0 mean in group 1
443.6667 1231.0779
Given that population density is a continuous variable, we can use a
t-test to look at differences in means and a
ks.test for meaningful differences in the distribution
between observed and unobserved polling stations.
All these stats show that there is an imbalance between observed and unobserved polling stations. That is, they treatment (monitoring) was not randomly received with regard to population density and urban areas. This is a very concerning problem as we know from various studies that urban and rural demographics are different, particularly when it comes to political predisposition. The problem here is that what we might think to be the treatment effect is actually just this difference we pick up in our results due to selection bias.
Exercise 17: Randomise the assignment to treatment. Use (i) complete random assignment and (ii) stratified randomisation.
Since randomisation was not, in fact, successful in the natural
setting, let’s ourselves create a variable for random treatment
assignment. We assume that 50% of all polling stations can be
observed.
Note: There are two ways to do this:
- You can use either the rbinom function to randomly
assign a dummy variable, which follows the following syntax:
`rbinom(n, size, prob), where n is the number
of observations, size is the number of trials per
observation and prob the porbability of success (or a
positice result)
- Or you use the simple_ra function from the
randomizrpackage. You only need to specify arguments for
n and probability.
hyde2007$random_assignment <- rbinom(n=1764, size=1, prob=0.5)
table(hyde2007$random_assignment)
0 1
881 883
# or
library(randomizr)
hyde2007$random_assignment <- simple_ra(N = 1764, prob = 0.5)
table(hyde2007$random_assignment)
0 1
892 872
randomizr
package]
The randomizr package includes various options for
randomisation. You can easily achieve block randomisation or even
randomise the assignment of various treatments. These are just a few
examples:
hyde2007$multiarm_assignment <- complete_ra(N = 1764, prob_each = c(0.2, 0.3, 0.5), conditions = c("Treatment 1", "Treatment 2", "Control")) ## multiple treatments
table(hyde2007$multiarm_assignment)
Treatment 1 Treatment 2 Control
352 530 882
hyde2007$block_assignment <- block_ra(blocks = hyde2007$urban,
prob_each = c(.1, .1, .8))
table(hyde2007$block_assignment)
T1 T2 T3
176 177 1411
Assume now that the OSCE has been made aware of this problem regarding
rural areas. For some reason, urban polling stations are more likely to
be monitored than rural ones. For their next assignment, they want to
make sure rural polling stations are just as likely to be observed as
urban ones. To have enough resources to ensure that every single polling
station that is assigned to be observers will in fact be observed, they
reduce the overall number of polling stations to 30% but use stratified
randomisation.
Note: The easiest way to do this is using the strata_rs
command from the randomizr package. You just need to
specify the strata and prob
arguments.
#install.packages("randomizr")
library(randomizr)
hyde2007$stratified <- strata_rs(strata = hyde2007$urban, prob = .3)
This should be much better now. Nonetheless, we should make sure randomisation was successful - using random assignment as well as stratified randomisation - to avoid working with flawed data.
# Complete Random Assignment Balance Checks
t.test(hyde2007$voterspersqkm~hyde2007$random_assignment, data=hyde2007)
Welch Two Sample t-test
data: hyde2007$voterspersqkm by hyde2007$random_assignment
t = -1.8259, df = 1749.8, p-value = 0.06804
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-239.255679 8.556565
sample estimates:
mean in group 0 mean in group 1
720.5368 835.8863
prop.test(table(hyde2007$urban, hyde2007$random_assignment))
2-sample test for equality of proportions with continuity correction
data: table(hyde2007$urban, hyde2007$random_assignment)
X-squared = 1.4462, df = 1, p-value = 0.2291
alternative hypothesis: two.sided
95 percent confidence interval:
-0.01811604 0.07802379
sample estimates:
prop 1 prop 2
0.5222930 0.4923391
# Stratified Randomisation Balance Checks
t.test(hyde2007$voterspersqkm~hyde2007$stratified, data=hyde2007)
Welch Two Sample t-test
data: hyde2007$voterspersqkm by hyde2007$stratified
t = -0.26094, df = 993.18, p-value = 0.7942
alternative hypothesis: true difference in means between group 0 and group 1 is not equal to 0
95 percent confidence interval:
-153.6756 117.6033
sample estimates:
mean in group 0 mean in group 1
772.1488 790.1850
prop.test(table(hyde2007$urban, hyde2007$stratified))
2-sample test for equality of proportions with continuity correction
data: table(hyde2007$urban, hyde2007$stratified)
X-squared = 1.0575e-29, df = 1, p-value = 1
alternative hypothesis: two.sided
95 percent confidence interval:
-0.04302202 0.04490877
sample estimates:
prop 1 prop 2
0.7006369 0.6996936
This looks good - actually, it couldn’t be better: look at the
p-value of 1! Assignment to treatment is now balanced.
Let’s have a look at the regional allocation of election observers
nonetheless.
aggregate(hyde2007[, c("mon_voting", "random_assignment", "stratified")], list(hyde2007$subregion), mean)
Group.1 mon_voting random_assignment stratified
1 Abovyan 0.4814815 0.3703704 0.4444444
2 Ajapnyak 0.5897436 0.6153846 0.4615385
3 Alaverdi 0.3200000 0.5800000 0.3200000
4 Anrashat 0.7142857 0.4761905 0.2380952
5 Arabkir 0.6818182 0.5909091 0.4090909
6 Ararat 0.6666667 0.6666667 0.1666667
7 Armavir 0.3600000 0.6000000 0.4400000
8 Arshaluys 0.2812500 0.4375000 0.3125000
9 Artik 0.4000000 0.4000000 0.3250000
10 Ashtarak 0.2812500 0.5781250 0.2343750
11 Avan 0.5238095 0.7619048 0.2857143
12 Ayntap 0.5333333 0.5333333 0.3666667
13 Berd 0.4186047 0.4651163 0.3255814
14 Byureghaven 0.3611111 0.3333333 0.2777778
15 Charentsavan 0.2222222 0.4814815 0.4074074
16 Eghvard 0.6153846 0.3076923 0.3846154
17 Erebuni 0.7692308 0.5192308 0.3076923
18 Gavar 0.2500000 0.4285714 0.3571429
19 Goris 0.3893805 0.5044248 0.3362832
20 Gyumri 0.1676301 0.4913295 0.3121387
21 Hrazdan 0.5862069 0.4827586 0.1724138
22 Ijevan 0.1351351 0.4864865 0.2702703
23 Kanaker-Zeytun 0.3571429 0.3095238 0.1428571
24 Kapan 1.0000000 0.0000000 0.0000000
25 Kentron 0.8600000 0.6000000 0.2400000
26 Malatia-Sebastia 0.7777778 0.4722222 0.2777778
27 Martuni 0.4074074 0.2962963 0.2222222
28 Masis 0.3793103 0.5517241 0.3448276
29 Myasnikyan 0.3000000 0.4250000 0.2750000
30 Nor-Nork 0.8333333 0.4791667 0.2708333
31 Parakar 0.4062500 0.3750000 0.3125000
32 Sevan 0.3023256 0.5116279 0.1860465
33 Shengavit 0.7142857 0.5714286 0.3392857
34 Talin 0.2753623 0.4782609 0.3043478
35 Tashir 0.2222222 0.4629630 0.2777778
36 Vanadzor 0.3934426 0.4672131 0.2868852
37 Vardenis 0.3400000 0.5600000 0.2600000
38 Vargharshapat 0.6666667 0.3750000 0.4583333
39 Vedi 0.4074074 0.4814815 0.2962963
40 Yeghegnadzor 0.3750000 0.5500000 0.2000000
It looks like the problem with first round monitoring is not merely a
concentration in urban areas, but an overproportionate presence of
monitors in the capital, Yerevan!
Arabkir, Erebuni, Kentron, Malatia-Sebastia, Nor-Nork, Shengavit
are all districts of Yerevan and vastly overrepresented. The table also
shows that this problem would not have appeared if observers had been
randomly assigned. This is a good example that context really matters
for natural experiments!
Make sure you will be using the osce_assignment variable
for the next parts of the lab. It provides a fictional example of
potential assignment by the OSCE, allowing us to work with the same
data.
One of the reasons for the imbalance between rural and urban polling
station could be that observers did not observe their assigned polling
stations. In that case, one-sided non-compliance would
be a problem. Polling stations that were supposed to be monitored were,
in fact, not monitored on election day. Let’s quickly have a look at the
compliance status and the broad picture. Let’s look again at the
contingency table that you did using the osce_assignment
and the mon_voting_R2 variables.
In experiments with one-sided compliance units or subjects can be divided into two groups: Compliers and Never takers . However, in the case of two-sided non-compliance there are subjects that be divided into further groups adding the so-called are called Always takers . These are subjects/units that regardless of their treatment assignment, they always end up taking the treatment.
Tricky Exercise 18: Generate again a contingency table using
the osce_assignment and the mon_voting_R2
variables. Calculate the proportion of Always
takers.
#Let's remember the compliance status [1] 883 [2]258 [3] 0 [4]623
tab.1 <- table(hyde2007$osce_assignment, hyde2007$mon_voting_R2)
tab.1
0 1
0 883 0
1 258 623
#Always-Takers
# Under one-sided non-compliance there are no Always takers, so this should be zero
# 0 / (883 + 0) = 0
at <- tab.1[3]/(tab.1[1] + tab.1[3])
at
[1] 0
As pointed out before, there are no Always takers in experiments with one-sided compliance. If there were Always takers , we would expect some of polling stations assigned to control (row = 0) being monitored (column = 1).
We also have the so-called Never takers . These are units that never take the treatment regardless of whether they are assigned to treatment group or control group.
Exercise 19: Calculate the proportion of Never takers in the study. Use the same contingency table as before. Hint: Look at the number of units within those assigned to treatment, and in particular the number of polling stations that were not monitored.
tab.1
0 1
0 883 0
1 258 623
#Never Takers
# 258/(258 + 623) = 0.29
# This means that from those polling stations that were assigned to treatment, 29% will not be monitored
nt <- tab.1[2]/(tab.1[2] + tab.1[4])
nt
[1] 0.292849
We can calculate the proportion of Never
takers by looking the units within the treatment group. The
number of Never takers are
those polling stations assigned to treatment, but not ending up taking
the treatment. This is equal to 258. Then, the proportion
is equal to 258 divided by the total number of polling
stations assigned to the treatment 258 + 623 = 881. We
cannot identify the Never takers
in the control group, as we cannot disentangle whether units are
not taking treatment because they are Compliers or because in fact they
are Never takers .
Finally, we have the so-called Compliers . These are units that follow whatever their assignment is (if assigned to treatment, they do take the treatment; if assigned to control, they don’t). For example, if a polling station was assigned to be monitored, it ends up being monitored. Conversely, if a polling station was assigned to not be monitored, it ends up not being monitored.
Exercise 20: Calculate the proportion of Compliers in the study. Use the same contingency table as before. Hint: Look again at the units in the treatment group. In particular the number of polling stations that were monitored.
#Compliers (No Defiers By Assumption)
1 - at - nt
[1] 0.707151
So we have 623 units assigned to the treatment and ended
up being monitored. Thus, the number of compliers is equal to
623/(258+623) = 0.70. But we can also obtain the proportion
of Compliers by subtracting the proportion of Never
takers and Always takers
from 1.
A key assumption for identification is that we rule out the existence of the alleged Defiers. These units do the opposite of whatever their assignment is (if assigned to treatment, they don’t do the treatment; if assigned to control, they do the treatment).
Using the ITT is one possible solution to dealing with
non-compliance. Remember that the ITT describes intended assignments,
not actual treatment. In doing so,it does not simply discard
non-compliance.
Let’s start by calculating the ITT manually, using differences in
means. Remember that the ITT describes the difference in the outcome
between the units assigned to treatment and those which were
not. In other words, it is the change in expected outcomes when a unit
moves from an assigned control group to an assigned treatment
group:
\[ITT = E[Y_1~_i] -
E[Y_0~_i]\]
Note that you find this parameter also written as
ITT.y, which highlights its referring to the effect in the
outcome, y.
Exercise 21: Using the information on random assignment
(osce_assignment), calculate the Intent-to-Treat effect
(ITT) in Round 2 of the election.
There are two ways to calculate the ITT.y. First, the
difference in means of the outcome variable between the group of units
assigned to treatment, on the one hand, and the units assigned to the
control group, on the other hand, is equivalent to the
ITT.
#ITT or ITT.y
itt.y <- (mean(hyde2007$KocharianR2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$KocharianR2[hyde2007$osce_assignment==0], na.rm = TRUE))
itt.y
[1] -0.01544005
Second, you can run a bivariate OLS regression. As you learned last week, a the coefficient of a bivariate OLS is equivalent to the difference in means if the independent variable is binary.
itt_fit <- lm(KocharianR2 ~ osce_assignment, data= hyde2007)
summary(itt_fit)
Call:
lm(formula = KocharianR2 ~ osce_assignment, data = hyde2007)
Residuals:
Min 1Q Median 3Q Max
-0.69322 -0.10448 -0.00248 0.11818 0.32222
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.693215 0.005439 127.456 <2e-16 ***
osce_assignment -0.015440 0.007694 -2.007 0.0449 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.1615 on 1761 degrees of freedom
(1 observation deleted due to missingness)
Multiple R-squared: 0.002282, Adjusted R-squared: 0.001715
F-statistic: 4.027 on 1 and 1761 DF, p-value: 0.04492
The results are the same, the ITT is
about-0.015. The difference in the outcome between polling
stations assigned to treatment and those that weren’t amounts to
1.5%points in the incumbent’s vote share. Does this go
together with the reported results?
If there is no non-compliance and assignment to treatment perfectly
predicts receiving the treatment, the ITT is equivalent to
the ATE. In our natural experiment, however, many polling
stations that should have been observed were not. Accordingly, the
ITT.y underestimates any true treatment effect. If we don’t
care about treatment effects but are only interested in the change in
outcome a certain programme or policy brings about, the ITT can be a
quantity of interest.
The ITT thus describes the intent-to-treat effect of assignment to
treatment (z) on the outcome (Y). Be aware that this
means the outcomes could be influenced either by the assignment to the
treatment or the actual treatment.
Let’s therefore now look at the intent-to-treat effect of assignment
to treatment (z) on treatment (d). This second ITT,
sometimes called `ITT.d, describes the the proportion of
subjects who are treated in the event that they are assigned to the
treatment group, minus the proportion who werre treated even if they had
instead been assigned to the control group:
\[ITT_D = E[d_i(1)] - E[d_i(0)]\]
Note that in the case of one-sided no compliance, e.g. the case
without always-takers as we are facing here, the ITT.D
reduces to \(E[d_i(1)]\). In our
analyses, we assume there were no always-takers, i.e. polling stations
that were not supposed to be monitored but were, in fact,
monitored.
Exercise 22: Calculate the Intent-to-Treat effect on
treatment (ITT.d) in Round 2 of the election.
Here, again, we have the same two options we used to estimate the
ITT. Both differences in means and a bivariate OLS will
provide the ITT.d. In the former case, it is the difference
in means of the treatment variable, which is equivalent to the share of
treated polling stations per group, between polling stations selected
for observation missions, on the one hand, on those that haven’t been
selected, on the other hand. In other words, we calculate the share of
monitored polling states over all stations assigned for monitoring and
subtract the share of monitored stations in the control group from the
former.
#ITT_D
itt.d <- (mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==0], na.rm = TRUE))
itt.d
[1] 0.707151
Equivalently, on OLS regression treatment on assignment to treatment will provide the same result:
## ITT_D using OLS
itt_d_fit <- lm(mon_voting_R2 ~ osce_assignment, data=hyde2007)
summary(itt_d_fit)
Call:
lm(formula = mon_voting_R2 ~ osce_assignment, data = hyde2007)
Residuals:
Min 1Q Median 3Q Max
-0.7071 0.0000 0.0000 0.2928 0.2928
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2.072e-15 1.083e-02 0.00 1
osce_assignment 7.072e-01 1.532e-02 46.15 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.3218 on 1762 degrees of freedom
Multiple R-squared: 0.5473, Adjusted R-squared: 0.547
F-statistic: 2130 on 1 and 1762 DF, p-value: < 2.2e-16
The results are the same, the ITT.d is
about0.71. In other terms, only 71% of all polling stations
that should have been observed were observed during the first round of
the elections.
We (and Hyde), however, do not care very much about the differences
in outcomes or the ITT itself. The aim of this natural experiment is
finding out if electoral monitoring significantly reduces fraud - we
want to identify the treatment effect. As stated above, the
ITT is not equivalent to the ATE due to
non-compliance.
In this case, the complier average causal effect
(CACE) can be estimated, which corresponds to the
ATE among compliers:
\[CACE = \frac{ITT_Y}{ITT_D} =
\underbrace{E[(Y_i(d=1) -
Y_i(d=0))}_\text{ATE}|\underbrace{d_i(1)=1}_\text{Among
Compliers}]\]
Note that the CACE is sometimes referred to as local average
treatment effect (LATE).
Exercise 23: Calculate the CACE using the ITT.y
and ITT.d
You can simply divide the ITT by the ITT.D
or use the calculations for both quantities and merge them into a single
formula.
#CACE in one line: ITT_Y divided by ITT_D
(mean(hyde2007$KocharianR2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$KocharianR2[hyde2007$osce_assignment==0], na.rm = TRUE))/(mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==1], na.rm = TRUE) - mean(hyde2007$mon_voting_R2[hyde2007$osce_assignment==0], na.rm = TRUE))
[1] -0.02183417
cace <- itt.y/itt.d # This should be the same!
print(cace)
[1] -0.02183417
We can also calculate the significance of the CACE. To do so, we calculate the standard errors of the CACE (this is in fact, an approximation): \(SE(CACE) \approx SE(ITT.Y) / ITT.D\)
se_cace <- summary(itt_fit)$coefficients[4]/itt_d_fit$coefficients
print(se_cace)
(Intercept) osce_assignment
-3.712522e+12 1.088013e-02
Finally, you can then calculate the p-value as we did last week:
t_cace <- cace / se_cace
print(t_cace)
(Intercept) osce_assignment
5.881222e-15 -2.006793e+00
p <- (1 - pnorm(abs(t_cace), 0, 1)) * 2
p
(Intercept) osce_assignment
1.00000000 0.04477166
Exercise 23b: Calculate the CACE using a two-stage
least-squares (2SLS) estimator
As we will be seeing in a few weeks time, we can also use two-stage
least-square (2SLS) regressions to calculate the CACE. This
regression is frequently used in instrumental variable designs, but
don’t worry about that for now. All we need to know is that it can be
applied to identify the CACE, too.
The logic of the 2SLS estimator is that it involves two regressions.
in a first step, the (random!) assignment to treatment is used to
predict actual treatment. Assignment to treatment itself is assumed to
only influence the outcome through the reception of treatment and is,
thus, not part of the second equation. The effect of the treatment in
the second equation is estimated using the predicted values for
receiving the treatment.
While this seems complicated, what 2SLS does is addressing
endogeneity: It accounts for the possibility that receiving treatment
itself can be related to some confounder that also affects the
treatment.
The 2SLS estimator can be implemented relatively easily using the
ivreg command. The syntax is as follows:
ivreg(Outcome ~ Treatment|Assignment, data=).
#CACE via 2SLS
iv_reg <- ivreg(KocharianR2 ~ mon_voting_R2|osce_assignment, data=hyde2007)
summary(iv_reg)
Call:
ivreg(formula = KocharianR2 ~ mon_voting_R2 | osce_assignment,
data = hyde2007)
Residuals:
Min 1Q Median 3Q Max
-0.693215 -0.103031 -0.002715 0.118335 0.328619
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.693215 0.005436 127.528 <2e-16 ***
mon_voting_R2 -0.021834 0.010874 -2.008 0.0448 *
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.1614 on 1761 degrees of freedom
Multiple R-Squared: 0.003409, Adjusted R-squared: 0.002843
Wald test: 4.032 on 1 and 1761 DF, p-value: 0.0448
The estimated CACE is equivalent to the one from our manual estimation,
-0.0218. That is, the average treatment effect among
polling stations that were supposed to be observed and, in fact, were
observed corresponds to a reduction in the incumbent’s vote share of
2.2%-points.
Note that as compared to a naive OLS, this takes into account that
(non-random) non-compliance takes place instead of disregarding
non-compliance. Nor does it bulk together different effects as the
ITT does in case of non-compliance.
Exercise 24: Plot the estimated coefficients from (i) the
2SLS regression and (ii) the naive OLS.
There are many ways to do this. The most common package to plot
coefficients is coefplot, but depending the estimator used,
some adjustment might be required. Sometimes it can even be necessary to
manually retrieve coefficients and confidence intervals.
Note that the 2SLS regression increases the uncertainty around the
point estimate as compared to the OLS regression.
The modelplot command is also a powerful tool to plot
your regression output:
#install.packages("modelsummary")
library("modelsummary")
# OLS
m_ols <- lm(KocharianR2 ~ mon_voting_R2, data=hyde2007)
# Combining OLS and IV to a list of models
m_list <- list(OLS = m_ols, IV = iv_reg)
msummary(m_list)
| OLS | IV | |
|---|---|---|
| (Intercept) | 0.693 | 0.693 |
| (0.005) | (0.005) | |
| mon_voting_R2 | −0.020 | −0.022 |
| (0.008) | (0.011) | |
| Num.Obs. | 1763 | 1763 |
| R2 | 0.003 | 0.003 |
| R2 Adj. | 0.003 | 0.003 |
| AIC | −1423.1 | |
| BIC | −1406.7 | |
| Log.Lik. | 714.547 | |
| F | 6.085 | |
| RMSE | 0.16 |
# Plotting the Coefficients
modelplot(m_list, conf_level = 0.95, coef_omit = "Intercept")
# You can also customise the plot using the ggplot syntax:
modelplot(m_list, conf_level = 0.95, coef_omit = "Intercept", coef_rename = c("mon_voting_R2"="Monitoring (R2)")) +
scale_color_brewer(type = 'qual') +
ggtitle("Coefficient Plot") +
xlab("Coefficients and 95% Confidence Intervals") +
theme_classic()